As mentioned in #273 (comment) 2MiB seems quite large (and won't work with IPNS over PubSub). If data is ever super large it could probably just point at a CID or another IPNS record instead.

Some ways it might be reasonable to establish a magic number here if we don't want to just spitball a number (which is also doable)

• What do records in the network currently look like? (e.g. use Hydras to analyze stored records)
• Look at something like a "maximum CID size" and add some more to it (e.g. double it). The indexer folks could probably give some insight here as would the related issues on this topic. Blocking on people committing to a maximum CID size seems unwise since that could take a while/not happen so we'd have to guesstimate here
• What's the longest path we've seen show up on ipfs.io
• Max public key size
  • Probably a 4096 RSA key, but you could ask the libp2p folks.
  • If people have really large keys we could either change the protocol to go back to having people find them separately from the records, or keep their values smaller to tradeoff against their key size and push towards smaller key types

Aside: it might also be worth considering switching the wording here to be similar to how Bitswap has block sizes specified.

It's a bit of a different take, but could be useful to document (even just on GitHub) why we chose one approach over the other.

Things are more complicated: the introduction of the extensible IpnsEntry.data CBOR field means some people and services WILL start putting additional metadata there, and that means all bets and estimates will be off.

Still, we need to create some arbitrary headroom. It is easier to increase the limit than to decrease it.
I've switched to max to 10 kiB as it is closer to realistic needs, and adjusted the language to match bitswap one (see 1baf069).

Will try to gather some data from hydras, but in the meantime, @alanshaw / @vasco-santos do you have any metrics around record sizes on w3name? or thoughts on this matter?

Some quick numbers from hydras (records that are currently valid)

Total DynamoDB Records: 256697
  ipns: 12287
  pk:   244410

DynamoDB Record Value Sizes:
  avg:    299.62269913555673
  std:    45.88040769545226
  median: 299
  max:    2033
  min:    36

PK Records Key Types:
  rsa:       244403
  ed25519:   3
  secp256k1: 0
  ecdsa:     4

And just the extensible data field

IPNS 'IpnsEntry.Data' Size:
  avg:    139.27228998453873
  std:    9.81409166725822
  median: 141
  max:    182
  min:    91

10 KiB sounds ok, leaving some room for growth and experimentation without being blocked by implementations that follow this spec, but lmk if we should adjust.

Seems mostly reasonable as this copies from the libp2p peerID spec, however RSA keys might still be very common since they used to be the default in implementations like kubo (formerly go-ipfs) and js-ipfs. This is the same as in the libp2p world, but given we're not just referencing that spec directly figured I'd call it out here in case anyone comes looking.

It'd be nice if we could post some numbers here (on GitHub) coming from the Hydra nodes on the distribution of key types justifying that Ed25519 covers most usage.

Will try to get % usage of Ed25519, RSA, Secp256k1 and ECDSA and get back to this with numbers.

Some quick numbers from hydras (records that are currently valid)

Total DynamoDB Records: 256697
  ipns: 12287
  pk:   244410

DynamoDB Record Value Sizes:
  avg:    299.62269913555673
  std:    45.88040769545226
  median: 299
  max:    2033
  min:    36
IPNS Records Signatures:
  v1:     12287
  v2:     6892
  with pk: 913
PK Records Key Types:
  rsa:       244403
  ed25519:   3
  secp256k1: 0
  ecdsa:     4

ed25519 is used by default and inlined, "3" is probably someone writing own IPNS tool and embedding it the same way RSA - can be ignored.

“with pk” means IPNS records with IpnsEntry.pubKey present.
RSA is dominating here, and used in ~7% of cases.

I think Ed25519 being ~93% and RSA ~7% is a good argument to keep Ed25519 as MUST and change RSA to SHOULD (same as in PeerID spec).

I adjusted language in 41612da

Some DHT stats

We've looked at some stats from Hydra nodes (a small sample of recently valid records):

IPNS Records Signatures:
  v1:       9513
  v2:       5821
  with pk:  733 (RSA)
  nil data: 3692 (V1-only)

Every V2 record is also V1 (V1-only records have no data field, so their count is in "nil data").
This means for the sample size of 9513 we have ~61% of V2+V1 and ~38% V1-only.
We also looked at other time-ranges and V2 is growing, V1-only is decreasing – seems to follow the usual slow update curve we also see for Kubo itself.

On this upgrade strategy

It is reasonable to expect both clients and publishers will update over time,
but the act of updating a publisher should not introduce any record resolution regressions for legacy nodes.

To make this clear, I've documented backward compatibility policy in b926e8c

• A legacy publisher MUST always be able to update to the latest implementation
  of this specification without breaking record resolution for legacy consumers.
• A legacy consumer MUST always be able to resolve IPNS name, even when publisher
  updated to the latest implementation of this specification.

This policy ensures the spec is aligned with existing code that was created to ensure IPNS implementations in GO and JS are backward-compatible from the perspective of legacy consumers:

• both Kubo and js-ipfs produce V1+V2 records
  • This allows legacy go-ipfs <0.9.0 node to correctly resolve IPNS records published by modern Kubo 0.15+
  • Maintaining this contract is important, I've added it to this spec
• Old go-ipfs <0.9.0 node publishing legacy RSA+V1-only records should be able to upgrade to Kubo 0.16 and start publishing V1+V2 records, allowing for V2 resolution on modern clients (but not break V1-only for legacy ones).

does "resolve IPNS" name mean they should be able to validate the signature? Or just that they can read the record?
Isn't that already not the case since supporting all Key types isn't required there's already consumers that can't validate all signtures? And new key types (or signing prefixes) would invalidate old consumers?

does "resolve IPNS" name mean they should be able to validate the signature?

@paulgmiller Yes, IPNS resolution includes signature verification step: records that have invalid signatureV2 are ignored by implementations like Kubo or js-ipfs.

And new key types (or signing prefixes) would invalidate old consumers?

The use case we are protecting here is preexisting IPNS client resolving preexisting IPNS name for mission-critical updates.

In that case, the key type will not change, records are published with the same key, and the old signature must be present next to the new one to ensure the name still resolves for the legacy client.

An example scenario is as follows:

• IoT device implemented minimal support for RSA and V1, and uses it for polling for new firmware updates.
• It retrieves record with V1+V2 signatures, validates V1 and fetches new firmware
• The new firmware has support for IPNS V2, and uses a new IPNS name for polling for updates (requires V2 signatures, and can also use a different key type like Ed25519).